Comprehension Modeling and AI-Sourced Student Content Recommendations

ABSTRACT

A system generates a training dataset based on historical consumption information and historical comprehension information of historical users, and uses the training dataset to train a machine-learned model to predict a measure of comprehension for a user consuming educational content. The system applies the machine-learned model to behaviors of a target user to determine a target measure of comprehension for target educational content, identifies one or more characteristics of the target educational content, applies a content identification model to identify supplemental educational content, and generates an educational content interface to present the supplemental educational content to the target user. In some examples, the system trains the machine-learned model to predict a collective measure of comprehension for a set of users consuming educational content, identifies a set of supplemental educational content, and generates a teacher interface to present the set of supplemental educational content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims a benefit of U.S. Provisional Application No.63/356,301, filed Jun. 28, 2022, all of which is incorporated byreference herein in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to content recommendation and,particularly, comprehension modeling and AI-sourced contentrecommendations.

BACKGROUND

Online learning has become increasingly prevalent. Specifically, onlineeducation platforms offer numerous advantages, such as accessibility,flexibility, and a wide range of courses. However, such platforms alsocome with certain problems, for instance because they often lack thesame level of personal interaction found in traditional classrooms. Theabsence of face-to-face interaction can make it difficult for studentsto engage with instructors and fellow students, ask questions, andreceive immediate feedback. This can result in a sense of isolation andreduced motivation to participate actively in the learning process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example education systemenvironment, in accordance with an embodiment.

FIG. 2 is a block diagram illustrating components of a content server,in accordance with an embodiment.

FIG. 3A is a flowchart depicting a computer-implemented process forproviding content recommendations, in accordance with an embodiment.

FIG. 3B is a flowchart depicting a computer-implemented process forproviding content recommendations, in accordance with an embodiment.

FIG. 4A illustrates an example user interface, in accordance with anembodiment.

FIG. 4B illustrates another example user interface, in accordance withan embodiment.

FIG. 4C illustrates another example user interface, in accordance withan embodiment.

FIG. 5 illustrates another example user interface, in accordance with anembodiment.

FIG. 6A illustrates another example user interface, in accordance withan embodiment.

FIG. 6B illustrates another example user interface, in accordance withan embodiment.

FIG. 6C illustrates another example user interface, in accordance withan embodiment.

FIG. 6D illustrates another example user interface, in accordance withan embodiment.

FIG. 7 is a block diagram illustrating components of an example machineable to read instructions from a machine-readable medium and executethem in one or more processors (or controllers), in accordance with anexample embodiment.

The figures depict various embodiments for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdiscussion that alternative embodiments of the structures and methodsillustrated herein may be employed without departing from the principlesdescribed herein.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Configuration Overview

Disclosed are systems (as well as methods and computer program codestored on non-transitory computer readable mediums) configured toprovide a system that accesses historical consumption information andhistorical comprehension information of a set of historical users. Thesystem generates a training dataset based on the accessed historicalconsumption information and the historical comprehension information anduses the training dataset to train a machine-learned model to predict ameasure of comprehension for a user consuming educational content. Thesystem applies the machine-learned model to behaviors of a target userto determine a target measure of comprehension for each of a pluralityof portions of target educational content. The system identifies one ormore characteristics of a portion of the target educational content,applies a content identification model to the identified characteristicsof the portion of the target educational content to identifysupplemental educational content, and generates an educational contentinterface to present the supplemental educational content to the targetuser.

In some embodiments, the system accesses historical consumptioninformation and historical comprehension information of a set ofhistorical users. The system generates a training dataset based on theaccessed historical consumption information and historical comprehensioninformation and uses the training dataset to train a machine-learnedmodel to predict a collective measure of comprehension for a set ofusers consuming educational content. The system applies themachine-learned model to behaviors of a set of target users to determinea collective target measure of comprehension for each of a plurality ofportions of the target educational content. The system identifies one ormore characteristics of the portion of the target educational content,applies a content identification model to the identified characteristicsto identify a set of supplemental educational content, and generates ateacher interface to present the set of supplemental educationalcontent.

The disclosed configurations beneficially provide a system (and/or amethod) for asynchronous and synchronous teaching and learning whichprovides users with an interactive and enriched learning experience. Thesystem allows a teaching user (e.g., teach, professor, educator,faculty, etc.) from different institutions and disciplines to use theircourse materials to develop and deliver engaging and scalable courses tothose attending online, in-person or in a hybrid situation. A teachinguser may use the system to develop courses that involve organizing,manipulating and sequencing educational content and interactive widgetsinto study materials, such as pre-work material, live sessions,follow-on assignments, etc. The student users may organize, augment, andcreate their own student notes/feedbacks with similar interfaceelements. The student users may share their notes with other students(e.g., in a study group) or share their notes with the teaching user(e.g., as an assignment). The feedbacks and reactions can form loops. Assuch, the users can constantly engage and interact with each other withrespect to the course content, either synchronously or asynchronously.

Example System Configuration

Figure (FIG. 1 is a block diagram that illustrates an education systemenvironment 100, in accordance with an embodiment. The education systemenvironment 100 includes a content server 110, one or more contentstores 120A, 120B, one or more client devices 130A, 130B, and a network160. The entities and components in the education system environment 100communicate with each other through a network 160. In variousembodiments, the education system environment 100 includes fewer oradditional components. In some embodiments, the education systemenvironment 100 also includes different components. While some of thecomponents in the education system environment 100 is described in asingular form, the education system environment 100 may include one ormore of each of the components. Different client devices 130 may alsoaccess the content server 110 simultaneously. The client device 130 andthe content server 110 may include some or all of the components of acomputing device such as one described with FIG. 7 and an appropriateoperating system.

In an example embodiment, the content server 110 may be a computingsystem that provides educational content to users. The content server110 may use a comprehension model to determine a measure ofcomprehension for a user consuming educational content. In someembodiments, the content server 110 may apply the comprehension model tobehaviors of a target user to determine a target measure ofcomprehension for target educational content. In response to the targetmeasure of comprehension being below a threshold, the content server 110may use a content identification model to identify supplementaleducation content to improve the user's comprehension on the targeteducational content. In some embodiments, the content server 110 may usea comprehension model to determine a collective measure of comprehensionfor a set of users consuming educational content and identify a set ofsupplement educational content for the set of users to improve acollective measure of comprehension on the target educational content.In some embodiments, the content server 110 may generate a userinterface to present the supplemental educational content to the users.In some embodiments, the content server 110 may collect data relatedcontent consumption behaviors and measures of comprehension and presentlearning analytics based on the collected data in a user interface forthe user to review.

The content store 120 stores various files and data of the contentserver 110. In some embodiments, the content store 120A is provided bycontent server 110. In some the content store 120B is external to thecontent server 110 and includes one or more computing devices thatinclude memory or other storage media for storing various files anddata. The data stored in the content store 120 includes a variety ofeducational content. Educational content may refer to any content thatis used to engage, inspire and inform users to learn, for example,teacher lectures, seminar presentation, peer student's notes, steamingvideos, etc. The educational content may include a variety of formats,such as, text, PDF, E-book (ePub format), video clip, streaming video,audio clip, streaming audio, image(s), rich text format (RTF) content,PowerPoint/Keynote, Word doc/Pages doc, Google docs, web site, datasource/data feed, HTML, 3D object, downloadable files, etc.

A user may enter user input via a client device 130. Client devices 130can be any personal or mobile computing devices such as smartphones,tablets, notebook computers, laptops, desktop computers, andsmartwatches as well as any home entertainment device such astelevisions, video game consoles, television boxes, and receivers. Theclient device 130 can present information received from the contentserver 110 to a user, for example in the form of user interfaces. Insome embodiments, the client device may be a student device 130A,operated by a student user; alternatively, the client device may be ateacher device 130B, operated by a teaching user. In some embodiments,the content server 110 may be stored and executed from the same machineas the client device 130.

A client device 130 includes one or more applications 142 and interfaces144 that may display visual elements of the applications 142. The clientdevice 130 may be any computing device. Examples of such client devices130 include personal computers (PC), desktop computers, laptopcomputers, tablets (e.g., iPADs), smartphones, wearable electronicdevices such as smartwatches, or any other suitable electronic devices.

The application 132 is a software application that operates at theclient device 130. In one embodiment, an application 132 is published bythe party that operates the content server 110 to allow clients tocommunicate with the content server 110. In various embodiments, anapplication 132 may be of different types. In one embodiment, anapplication 132 is a web application that runs on JavaScript and otherbackend algorithms. In the case of a web application, the application132 cooperates with a web browser to render a front-end interface 134.In another embodiment, an application 132 is a mobile application. Inyet another embodiment, an application 132 may be a software programthat operates on a desktop computer that runs on an operating systemsuch as LINUX, MICROSOFT WINDOWS, MAC OS, or CHROME OS.

An interface 134 is any suitable interface for a client to interact withthe content server 110. The client may communicate to the application132 and the content server 110 through the interface 134. The interface134 may take different forms. In one embodiment, the interface 144 maybe a web browser such as CHROME, FIREFOX, SAFARI, INTERNET EXPLORER,EDGE, etc. and the application 132 may be a web application that is runby the web browser. In one embodiment, the interface 134 is part of theapplication 132. For example, the interface 134 may be the front-endcomponent of a mobile application or a desktop application. In oneembodiment, the interface 134 also is a graphical user interface whichincludes graphical elements and user-friendly control elements. In oneembodiment, the interface 134 may display a graphical user interfacethat provides learning analytics of educational consumption behaviors.The interface 134 may further include one or more user interactiveelements so that a user may with the interface 134. In some embodiments,the interface 134 presents supplemental educational content to a user.In some embodiments, the interface 134 presents a set of supplementaleducational content to a teaching user for selection and presents theselected supplemental educational content to student users for review.Examples of the interface 144 are discussed in further detail below withreference to FIG. 4A to FIG. 6D.

The network 160 provides connections to the components of the systemenvironment 100 through one or more sub-networks, which may include anycombination of local area and/or wide area networks, using both wiredand/or wireless communication systems. In one embodiment, a network 160uses standard communications technologies and/or protocols. For example,a network 160 may include communication links using technologies such asEthernet, 802.11, worldwide interoperability for microwave access(WiMAX), 3G, 4G, Long Term Evolution (LTE), 5G, code division multipleaccess (CDMA), digital subscriber line (DSL), etc. Examples of networkprotocols used for communicating via the network 160 includemultiprotocol label switching (MPLS), transmission controlprotocol/Internet protocol (TCP/IP), hypertext transport protocol(HTTP), simple mail transfer protocol (SMTP), and file transfer protocol(FTP). Data exchanged over a network 160 may be represented using anysuitable format, such as hypertext markup language (HTML), extensiblemarkup language (XML), JavaScript object notation (JSON), structuredquery language (SQL). In some embodiments, some of the communicationlinks of a network 160 may be encrypted using any suitable technique ortechniques such as secure sockets layer (SSL), transport layer security(TLS), virtual private networks (VPNs), Internet Protocol security(IPsec), etc. The network 160 also includes links and packet switchingnetworks such as the Internet. In some embodiments, a data store belongsto part of the internal computing system of a server (e.g., the datastore 120 may be part of the content server 110). In such cases, thenetwork 160 may be a local network that enables the server tocommunicate with the rest of the components.

Example Content Server Components

FIG. 2 is a block diagram illustrating components of a content server110, in accordance with an embodiment. The content server 110 includes abehavior monitoring module 210, a comprehension engine 220, a contentidentification engine 230, an interface 240, a communication module 250,a user database 260, model(s) 270, training dataset(s) 280, and acontent store 120. In various embodiments, the content server 110 mayinclude fewer or additional components. The content server 110 also mayinclude different components. The functions of various components may bedistributed in a different manner than described below. Moreover, whileeach of the components in FIG. 2 may be described in a singular form,the components may present in plurality. The components may take theform of a combination of software and hardware, such as software (e.g.,program code comprised of instructions) that is stored on memory andexecutable by a processing system (e.g., one or more processors).

The behavior monitoring module 210 collects data associated with auser's educational content consumption behaviors, e.g., user behaviorsrelated to the user's consumption of educational content. In someembodiments, the educational content consumption behaviors may includeparameters such as, reading rates, pause time, number of re-read times,delays in content consumption, highlighting, types of highlighting,highlight coverage, time of day, switch outs, switch duration, and/orsimilar parameters that describe a user's action/performance during theconsumption of educational content. For example, a user may tag aportion of educational content with “unclear,” “important,”“interesting,” and/or “debatable.” The behavior monitoring module 210 iscapable of receiving communications from the client device 130 about auser's actions on and/or off the education system environment 100. Insome embodiments, the behavior monitoring module 210 may record theeducational content consumption behaviors in real time as the userconsumes the content; alternatively, the user's educational contentconsumption behaviors may be monitored periodically. For example, thebehavior monitoring module 210 may record the reading time after a usercompletes reading of a whole chapter. In another example, the behaviormonitoring module 210 may collect the reading rate per page, or recordan average reading rate for a whole chapter. In some embodiments, thebehavior monitoring module 210 may aggregate the monitored user'sbehavior and present it to the user for review. Further detail isdiscussed below with reference to FIG. 5 . The behavior monitoringmodule 210 stores the collected educational content consumptionbehaviors in the user database 260. In some embodiments, the collectededucational content consumption behaviors may be stored in the trainingdatasets 280 for training the models 270.

In some embodiments, the behavior monitoring module 210 may obtaincollective educational content consumption behaviors for a set of users.The set of users may consume the same educational content, for example,students registering the same class, audiences attending the samelecture, etc. In some instance, the set of users may be determined basedon the educational content the users consume, time and/or location ofthe users consume the educational content, etc. The collectiveeducational content may be stored in the user database 260, and/or usedas the training datasets 280 for training the models 270.

The comprehension engine 220 determines a measure of comprehension for auser consuming educational content. The measure of comprehensiondescribes a level of comprehension when a user consumes educationalcontent. In some embodiments, the educational content may include one ormore portions, and the comprehension engine 220 may determine themeasure of comprehension for each of the one or more portions. In oneembodiment, the comprehension engine 220 may determines the measure ofcomprehension in real time as the user consumes the content; in anotherembodiment, the measure of comprehension may be determined periodically.For example, the comprehension engine 220 may determine the measure ofcomprehension after a user consumes one page of educational content;alternatively, the comprehension engine 220 may determine an averagemeasure of comprehension after the user finishes a whole chapter.

The comprehension engine 220 may determine a measure of comprehensionbased on behaviors of the user as the user consumes the educationalcontent. For example, for a user consumes a portion of educationalcontent, more switch out times may relate to a lower concentration thanless switch out times. In some embodiments, the comprehension engine 220may compare the behaviors of user to statistical behaviors of historicalusers to determine the measure of comprehension. For example, for aportion of educational content, an average duration time of consumption,e.g., determined based on historical user behaviors, may be about 10minutes. If a user takes more than 30 minutes or less than 1 minute tocomplete the portion of educational content, the comprehension enginemay determine the user's comprehension is low.

In some embodiments, the comprehension engine 220 applies one or moremodels 270 to determine a measure of comprehension for a user. Themodels 270 may predict a measure of comprehension for a user based onthe analysis performed on historical data. In one implementation, thecomprehension engine 220 applies a machine-learned comprehension modelto behaviors of a user consuming educational content to determine ameasure of comprehension for the user. The comprehension model may betrained with training datasets 280, which includes historicalconsumption information and historical comprehension information of aset of historical users. In some embodiments, the historical consumptioninformation may include information that describes historicaleducational content consumption behaviors of the set of the historicalusers. The historical comprehension information may include historicalmeasures of comprehension of historical educational content for the setof historical users. In some embodiments, the measure of comprehensionof educational content may include one or more of test scores, userevaluations of comprehension, types of highlighting, post-comprehensionquiz results. For example, a historical measure of comprehension may berelated to a student's performance on pop quizzes after consuming theeducational content, and/or a student's highlight as“unclear/confusing.”

In some embodiments, the comprehension engine 220 determines acollective measure of comprehension for a set of users consumingeducational content. The collective measure of comprehension describes alevel of comprehension for the set of users who consumes educationalcontent. In some embodiments, the educational content may include one ormore portions, and the comprehension engine 220 may determine acollective measure of comprehension for each of the one or more portionsof the educational content. In some embodiments, the comprehensionengine 220 may apply a machine-learned comprehension model to behaviorsof the set of users, for example, collective educational contentconsumption behaviors, to predict the collective measure ofcomprehension for the corresponding educational content.

The content identification engine 230 may identify one or morecharacteristics of the target educational content. The characteristicsof the educational content may refer to the content, perspectives,formats, lengths, publication time, authors, etc. For example, lectureslides on “Introduction to Biology” can be supplemented with a video“Introduction to Biology” which is about the same content but indifferent formats. In another example, a supplemental educationalcontent for a textbook on “Data analysis” from Author A be a series ofonline animations showing processes of data analysis. Thesecharacteristics of educational content may be stored in the contentstores 120.

The content identification engine 230 identifies supplementaleducational content based on identified characteristics of the targeteducational content. The supplemental educational content may beeducational content that have similar characteristics to the targeteducational content and can be used to supplement the target educationalcontent to improve the user's comprehension. In one implementation, thecontent identification engine 230 may access content that are stored inthe content store 120. The content identification engine 230 mayidentify a set of candidate content from the stored content anddetermine one or more of the set of candidate content as thesupplemental content for the target educational content. In someembodiments, the supplemental educational content and the targeteducational content were consumed by the same historical users, and thecontent identification engine 230 may determine the supplementaleducational content based on user feedbacks. In some other embodiments,the content identification engine 230 may determine the supplementaleducational content based on historical user data, such as educationalcontent consumption behaviors and/or measures of comprehension ofeducational content. In still other embodiments, the contentidentification engine 230 may determine the supplemental educationalcontent based on statistical data of a set of users.

In one instance, the content identification engine 230 applies one ormore models 270 to identify the supplemental educational content relatedto the target educational content. In some embodiments, the models 270may include a machine-learned content identification model. The contentidentification model may be trained with a training dataset 280 whichincludes historical educational content consumed by historical users,and the historical educational content includes one or more targeteducational content and supplemental educational content associated witheach target educational content consumed by the historical users. In oneimplementation, the content identification model is configured topredict a likelihood that the supplemental educational content willimprove a user's measure of comprehension for the target educationalcontent. In another implementation, the content identification model maybe configured to predict a relatedness measure that describes therelatedness between the supplemental educational content and the targeteducational content.

The content identification engine 230 receives the output of the contentidentification model and identifies the supplemental content based onthe output. For example, for a content identification model thatpredicts likelihood of improvement on the user's measure ofcomprehension, the content identification engine 230 may rank thelikelihood for each of the candidate content and select the candidatecontent with the highest likelihood as the supplemental content for thetarget educational content. Alternatively, the content identificationengine 230 may determine a threshold of the likelihood, and select oneor more candidate content of which likelihoods are higher than thethreshold, as the supplemental educational content. Similarly, for acontent identification model that predict a relatedness measure, thecontent identification engine 230 may select the supplemental contentbased on the ranking of the relatedness measure or using a thresholdrelatedness measure to select one or more candidate content as thesupplemental educational content.

In some embodiments, the content identification engine 230 may identifypersonalized supplemental educational content for a user. For example,the content identification model may be trained with a training dataset280 that includes historical educational content consumption behaviorsand corresponding historical measures of comprehension of educationalcontent for the user. In this way, the content identification model maypredict a supplemental educational content that suits the specific user,e.g., most likely to improve the user's measure of comprehension. Forexample, a user who is more efficient with listening to lectures, thecontent identification engine 230 may identify an audio record as asupplemental educational content to improve the user's measure ofcomprehension.

In some embodiments, the content identification engine 230 determines aset of supplemental educational content for a set of users consuming thetarget educational content. In one instance, the content identificationmodel may be trained with a training dataset 280 that includescollective educational content consumption behaviors and correspondingcollective measures of comprehension of educational content for the setof users. In one example, the content identification model may predict aset of supplemental educational content that suit for the set of users,e.g., most likely to improve the collective measure of comprehension forthe set of users. For example, the content identification engine 230 mayidentify a set of supplement educational content for a class of studentswho consume the same educational content, e.g., attending the sameclass.

In some embodiments, the target educational content may include one ormore portions, and the content identification engine 230 may identifysupplemental educational content related to each portion of the targeteducational content. In some embodiments, one or more portions of thetarget educational content are related to a low-comprehension dedicatedsection, and the content identification engine 230 may identifysupplemental educational content for the low-comprehension dedicatedsection.

In some embodiments, the content identification engine 230 may identifythe supplemental educational content in real time as the user consumesthe content, e.g., at the end of each page of the target educationalcontent. The content identification engine 230 may continuously updatethe supplemental educational content in real time. Alternatively, thecontent identification engine 230 may identify supplemental educationalcontent after the user finishes the consumption of the whole targeteducational content, e.g., at the end of the lecture, at the end of achapter, etc.

The interface 240 includes interfaces that are used to communicate withthe client devices 130. The interface 240 is in communication with theapplication 142 of a client device 130 and provides data to render theapplication 142. In one embodiment, the interface 240 provides a clientdevice 130 in the form of a graphical user interface (GUI) for users todisplay educational content consumption behaviors and supplementaleducational content. For example, the interface 240 may provideactionable learning analytics based on the educational contentconsumption behaviors to the users. In some embodiments, the interface240 may generate an educational content interface to present thesupplemental educational content to a target student user. In someembodiments, the interface 240 may generate an educational contentinterface (e.g., teacher interface) to present a set of supplementaleducational content to a target teaching user. The target teaching usermay select one or more of the set of supplemental educational contentvia the educational content interface. The interface 240 may generate astudent interface to present the selected supplemental educationalcontent to a set of target student users.

The interface 240 may receives user interactions/requests to modify theeducational content interface displayed on the client device. Forexample, the interface 240 may generate a student interface to presentthe selected portions of the supplemental educational content to each ofthe set of the target users. In another example, the interface 240 maypresent the selected portions of the supplemental educational content toone or more of the set of the target users, e.g., users with abelow-threshold measure of comprehension, users who prefer certain typesof educational content, etc. In yet another example, the interface 240may present the selected portion of the supplemental educational contentin a subsequent education session, for example, in a subsequentlesson/course, in a test/quiz, etc. In some embodiments, theconfiguration of the interfaces presented to the user may be customizedby the user, and/or pre-determined by a teaching user. Examples of theinterface 144 are discussed in further detail below with reference toFIG. 4A to FIG. 6D.

The communication module 250 transmits communication information betweenthe client devices 130. For example, the communication module 250receives a question from a first client device 130 on a portion of theeducational content and sends the question to a second client device130. The first client device 130 may be a student device that isoperated by a student user, and the second client device 130 may be ateacher device that is operated by a teaching user. The teacher devicemay send an answer to the question to the student device via thecommunication module 250. In some embodiments, the communication module250 may transmit questions, notifications, messages, feedbacks,comments, and the like among one or more client devices 130. In someembodiments, the communication module 250 may provide an interactiveinterface through which the users may message with each other andreceive notifications regarding the communication information related tothe educational content.

The models 270 includes various models accessed by the comprehensionengine 220 and the content identification engine 230. In someembodiments, the models 270 may include a comprehension model that isassessable by the comprehension engine 220 to predict a measure ofcomprehension for a user. In some embodiments, the comprehension modelmay predict a collective measure of comprehension for a set of usersconsuming educational content. In one implementation, the models 270 mayinclude a content identification model that is accessible by the contentidentification engine 230 to predict supplemental educational contentrelated to target educational content for a specific user. In anotherimplementation, the content identification model may predict a set ofsupplemental educational content for a set of users consuming the targeteducational content.

In some embodiments, the comprehension model and/or the contentidentification model may be a machine-learned model. For example, thecomprehension model may include a multi-feature linear regression model.In one implementation, the comprehension model may be trained usingsupervised model that uses data describing the educational contentconsumption behaviors and a historical user's comprehension of theeducational content. In some embodiments, a machine-learned model isassociated with an objective function, which generates a metric valuethat describes the objective goal of the training process. For example,the training intends to reduce the error rate of the model in generatingpredictions of the corresponding information in an upcoming time period.In such a case, the objective function may monitor the error rate of themachine-learned model. In some embodiments, the machine-learned modelincludes certain layers, nodes, kernels, and/or coefficients. Trainingof the machine-learned model includes iterations of forward propagationand backpropagation. Each layer in a neural network may include one ormore nodes, which may be fully or partially connected to other nodes inadjacent layers. In forward propagation, the neural network performs thecomputation in the forward direction based on outputs of a precedinglayer. The operation of a node may be defined by one or more functions.The functions that define the operation of a node may include variouscomputation operations such as convolution of data with one or morekernels, pooling, recurrent loop in RNN, various gates in LSTM, etc. Thefunctions may also include an activation function that adjusts theweight of the output of the node. Nodes in different layers may beassociated with different functions.

In some embodiments, the models 270 may be trained using datasets storedin the training datasets 280, and/or external data provided by athird-party data source. In some embodiments, the training datasets 280may store historical consumption information and historicalcomprehension information of historical users, and/or external dataprovided by a third-party data source. In some embodiments, the trainingdatasets 280 may be updated periodically, continuously, and/or in realtime. For example, a student's learning and comprehension may improveovertime, new students may add to a class, some students may drop aclass, etc. To maintain and improve the accuracy of the models 270, thetraining datasets 280 may be updated with the user's new information.For example, the newly collected educational content consumptionbehaviors and measure of comprehension may be added to the trainingdatasets 280. The models 270 may be retrained or fine-tuned with theupdated training datasets 280. In one example, the models 280 isretrained using the combined dataset of the original training data andthe new data. In another example, the models 280 may be updatedincrementally using the new data while leveraging the existing knowledgecaptured by the previous models. In some embodiments, the models 270 maybe continuously updated in real time as the user consumes theeducational content; alternatively; the models 270 may be updatedperiodically, such as, per day, per month, per semester, and/or afterthe consumption of each lecture, each chapter, etc.

Content Recommendation Process

FIG. 3A is a flowchart depicting a computer-implemented process 300 forproviding content recommendations, in accordance with an embodiment. Acomputer associated with the content server 110 includes a firstprocessor and first memory. The first memory stores a set of codeinstructions that, when executed by the first processor, causes thefirst processor to perform some of the steps described in the process300. Other entities may perform some or all of the steps in FIG. 3A. Thecontent server 110 as well as the other entities may include some or ofthe component of the machine (e.g., computer system) described inconjunction with FIG. 7 . Embodiments may include different and/oradditional steps, or perform the steps in different orders.

The content server 110 accesses 302 historical consumption informationof a set of historical users and historical comprehension information ofthe set of historical users. The historical consumption information maydescribe historical educational content consumption behaviors of the setof historical users. The historical comprehension information mayinclude a historical measure of comprehension of historical educationalcontent by the set of historical users. In some embodiments, theeducational content consumption behaviors include one or more of readingrates, pause time, number of re-read times, delays in contentconsumption, highlighting, types of highlighting, highlight coverage,time of day, switch outs, and switch duration. In some embodiments, themeasure of comprehension may include one or more of test scores, userevaluations of comprehension, types of highlighting, post-comprehensionquiz results. The historical consumption information and historicalcomprehension information of the set of historical users may be storeduser database 260. In some embodiments, the content server 110 maycollect the consumption information and historical comprehensioninformation of the set of historical users; alternatively, the contentserver 110 may access the consumption information and historicalcomprehension information that are provided by a third party.

The content server 110 generates 304 a training dataset based on theaccessed historical consumption information and the historicalcomprehension information. In some embodiments, the historicalconsumption information and historical comprehension information of theset of historical users may be stored in the training datasets 280 fortraining the models 270.

The content server 110 trains 306 a machine-learned model. Themachine-learned model may be a comprehension model that predicts ameasure of comprehension for a user who consumes educational content.The measure of comprehension describes a level of comprehension when auser consumes educational content. The content server 110 may determinethe measure of comprehension based on behaviors of the user as the userconsumes the educational content.

The content server 110 applies 308 the machine-learned model tobehaviors of a target user who consumes target educational content todetermine a target measure of comprehension for each of a plurality ofportions of the target educational content. In some embodiments, theeducational content may include one or more portions, and the contentserver 110 may determine the target measure of comprehension for each ofthe one or more portions. In one embodiment, the content server 110 maydetermines the target measure of comprehension in real time as the userconsumes the content; in another embodiment, the target measure ofcomprehension may be determined periodically. For example, the contentserver 110 may determine the target measure of comprehension after auser consumes one page of educational content; alternatively, thecontent server 110 may determine an average target measure ofcomprehension after the user finishes a whole chapter.

The content server 110 identifies 310 one or more characteristics of theportion of the target educational content. In some embodiments, thecontent server 110 may determine whether the target measure ofcomprehension for a portion of the target educational content is abelow-threshold target measure of comprehension. If the determinedtarget measure of comprehension is below a threshold, the content server110 may identify the characteristics for the portion of the targeteducational content to recommend a supplemental educational content tothe user to improve the user's comprehension associated with the portionof educational content. The characteristics of the educational contentmay refer to the content, perspectives, formats, lengths, publicationtime, authors, etc.

The content server 110 applies 312 a content identification model to theidentified characteristics of the portion of the target educationalcontent to identify supplemental educational content related to theportion of the target educational content. The content server 110identifies supplemental educational content based on identifiedcharacteristics of portion of the target educational content.

The content identification model may be trained with a training dataset280 which includes historical educational content consumed by historicalusers, and the historical educational content includes one or moretarget educational content and supplemental educational contentassociated with each portion of the target educational content consumedby the historical users. In one implementation, the contentidentification model is configured to predict a likelihood that thesupplemental educational content will improve a user's measure ofcomprehension for the portion of the target educational content. Inanother implementation, the content identification model may beconfigured to predict a relatedness measure that describes therelatedness between the supplemental educational content and the portionof the target educational content.

The content server 110 may identify a set of candidate content based onthe output of the content identification model. The content server 110may rank the set of candidate content based on, e.g., the likelihood ofimproving the user's measure of comprehension, the relatedness betweenthe portion of the target educational content and each of the set ofcandidate content. The content server 110 may select the candidatecontent with the highest ranking as the supplemental educational contentor select a set of candidate content of which ranking is above athreshold as a set of supplemental educational content.

In some embodiments, the content server 110 may identify thesupplemental educational content in real time as the user consumes thecontent, e.g., at the end of each page of the target educationalcontent. The content server 110 may continuously update the supplementaleducational content in real time. Alternatively, the contentidentification engine 230 may identify supplemental educational contentafter the user finishes the consumption of the whole target educationalcontent, e.g., at the end of the lecture, at the end of a chapter, etc.

The content server 110 generates 314 an educational content interface topresent the supplemental educational content to the target user. In someembodiments, the content server 110 may display the supplementaleducational content to the target user in real time as the target userconsumes the portion of the target educational content. Alternatively,the content server 110 may display the supplemental educational contentto the target user after the target user consumes the portion of thetarget educational content. In some embodiments, more than one portionof the target educational content may be associated with abelow-threshold target measure of comprehension, and these portions ofthe target educational content may be related to a low-comprehensiondedicated section. The content server 110 may identify secondsupplemental educational content related to a second portion of thetarget educational content associated with a below-threshold targetmeasure of comprehension. The content server 110 modifies theeducational content interface to add the second supplemental educationalcontent for display.

FIG. 3B is a flowchart depicting a computer-implemented process 350 forproviding content recommendations, in accordance with an embodiment. Acomputer associated with the content server 110 includes a firstprocessor and first memory. The first memory stores a set of codeinstructions that, when executed by the first processor, causes thefirst processor to perform some of the steps described in the process350. Other entities may perform some or all of the steps in FIG. 3B. Thecontent server 110 as well as the other entities may include some or ofthe component of the machine (e.g., computer system) described inconjunction with FIG. 7 . Embodiments may include different and/oradditional steps, or perform the steps in different orders.

The content server 110 accesses 352 historical consumption informationand historical comprehension information of a set of historical users.The historical consumption information may describe historicaleducational content consumption behaviors of the set of historicalusers. The historical comprehension information may include a historicalmeasure of comprehension of historical educational content by the set ofhistorical users. In some embodiments, the educational contentconsumption behaviors include one or more of reading rates, pause time,number of re-read times, delays in content consumption, highlighting,types of highlighting, highlight coverage, time of day, switch outs, andswitch duration. In some embodiments, the measure of comprehension mayinclude one or more of test scores, user evaluations of comprehension,types of highlighting, post-comprehension quiz results. The historicalconsumption information and historical comprehension information of theset of historical users may be stored user database 260. In someembodiments, the content server 110 may collect the consumptioninformation and historical comprehension information of the set ofhistorical users; alternatively, the content server 110 may access theconsumption information and historical comprehension information thatare provided by a third party.

The content server 110 generates 254 a training dataset based on theaccessed historical consumption information and historical comprehensioninformation. In some embodiments, the historical consumption informationand historical comprehension information of the set of historical usersmay be stored in the training datasets 280 for training the models 270.

The content server 110 trains 256 a machine-learned model which includesa comprehension model configured to predict a collective measure ofcomprehension for a set of users consuming educational content. Thecollective measure of comprehension describes a level of collectivecomprehension for the set of users when the set of users consumeeducational content. The content server 110 may determine the measure ofcomprehension based on behaviors of the set of users as the usersconsume the educational content.

The content server 110 applies 258 the machine-learned model tobehaviors of a set of target users consuming target educational contentto determine a collective target measure of comprehension for each of aplurality of portions of the target educational content.

The content server 110 identifies 360 one or more characteristics of theportion of the target educational content. In some embodiments, thecontent server 110 may determine whether the collective measure ofcomprehension for a portion of the target educational content is abelow-threshold collective target measure of comprehension. If thedetermined collective target measure of comprehension is below thethreshold, the content server 110 may identify the characteristics forthe portion of the target educational content to recommend a set ofsupplemental educational content to the set of target users to improvethe collective comprehension associated with the portion of educationalcontent.

The content server 110 applies 362 a content identification model to theidentified characteristics of the portion of the target educationalcontent to identify a set of supplemental educational content related tothe portion of the target educational content. The content server 110identifies the set of supplemental educational content based onidentified characteristics of portion of the target educational content.

The content identification model may be trained with a training dataset280 which includes historical educational content consumed by historicalusers, and the historical educational content includes one or moretarget educational content and supplemental educational contentassociated with each portion of the target educational content consumedby the historical users. In one implementation, the contentidentification model is configured to predict a likelihood that thesupplemental educational content will improve a collective measure ofcomprehension for the portion of the target educational content. Inanother implementation, the content identification model may beconfigured to predict a relatedness measure that describes therelatedness between the supplemental educational content and the portionof the target educational content.

In some embodiments, the content server 110 may identify the set ofsupplemental educational content in real time as the set of usersconsume the content, e.g., at the end of each page of the targeteducational content. The content server 110 may continuously update theset of supplemental educational content in real time. Alternatively, thecontent identification engine 230 may identify the set of supplementaleducational content after the user finishes the consumption of the wholetarget educational content, e.g., at the end of the lecture, at the endof a chapter, etc. In one implementation, the content server 110 maypresent the set of supplemental educational content in a subsequenteducation session, for example, in a subsequent lesson/course, in atest/quiz, etc.

The content server 110 generates 364 a teacher interface to present theset of supplemental educational content with the target educationalcontent. In some embodiments, a teaching user may input a selection ofone or more portions of the set of supplemental educational content viathe teacher interface. The content server 110 receives the selection ofone or more portions of supplemental educational content via the teacherinterface and generates a student interface to present the selected oneor more portions of supplemental educational content. In someembodiments, the content server 110 may present the selected one or moreportions of supplemental educational content to each of the set oftarget users. Alternatively, the content server 110 may present theselected one or more portions of supplemental educational content to oneor more target users, e.g., users with a below-threshold measure ofcomprehension, users who prefer certain types of educational content,etc.

Example User Interfaces

FIG. 4A illustrates an example user interface 400, in accordance with anembodiment. The user interface 400 is a graphical user interface thatmay be provided by the interface 240 of the content server 110. The userinterface 400 is displayed by the interface 134 of a client device 130.In some embodiments, the client device 130 may be a student device 130A,i.e., operated by a student user. In some embodiments, the userinterface 400 may be displayed by a webpage browser, a mobileapplication, etc.

As shown in FIG. 4A, the user interface 400 includes educational content402 and an interface element 410. The user interface 400 presents theeducational content 402 for a user to consume. The interface element 410includes a set of interactive elements, 412, 414, 416, and 418, whichallow the user to interact with the user interface 400. The interfaceelement 410 is associated with the educational content 402. Based on theuser's interaction with the interactive elements, the user interface 400may obtain educational content consumption behaviors of the user and/orthe measure of comprehension of the educational content 402.

Referring to FIG. 4B, which illustrates an example user interface 400,in accordance with an embodiment. Following the context of the exampledescribed in FIG. 4A, FIG. 4B shows the user interface 400 expands toprovide a second set of interactive elements, such as 422, 424, 426, and428. Each interactive element may be associated with a tag, for example,“unclear,” “important,” “interesting,” and “debatable.” By interactingwith an interactive element, a user may add the corresponding tag to theeducational content 402. The user interface 400 may expand to add theadditional interactive elements in response to the user interacts (e.g.,clicking, pressing, hovering, etc.) with an interactive element, e.g.,412, 414, 416, or 418. The two sets of interactive elements, e.g.,412-418 and 422-428, may be paired, e.g., the interactive element 412 ispaired with the interactive element 422. In one example, a user clicksthe interactive element 412 may result in adding a tag of “unclear” tothe educational content 402.

FIG. 4C illustrates an example user interface 400, in accordance with anembodiment. Following the context of the example described in FIG. 4Aand FIG. 4B, the user interface 400 presents learning analyticsassociated with the educational content. As shown in FIG. 4C, theeducational content may include one or more portions, e.g., 404, 406,and 408. Based on the user interaction with the user interface 400 asdiscussed in FIG. 4A and FIG. 4B, each portion of the educationalcontent may be associated with an interface element, e.g., 430, whichindicates the learning analytics associated with the portion ofeducational content. When a user interacts with the interface element430, the user interface 400 may provide a second interface element 440which presents detailed learning analytics. In some embodiments, thelearning analytics is associated with the educational contentconsumption behaviors and/or measures of comprehension for a set ofusers. As shown in FIG. 4C, the interface element 430 may indicate thatfor the portion of educational content 406, 20 users tagged it as“important,” 1 user tagged it as “interesting,” etc. In someembodiments, the interface element 430 may include users' comments,shown as “9 public note,” which is viewable to all of the set of users.In some embodiments, the interface element 430 may include a teachinguser's comment/feedback, such as “teaching plan.”

FIG. 5 illustrates an example user interface 500, in accordance with anembodiment. The user interface 500 is a graphical user interface thatmay be provided by the interface 240 of the content server 110. The userinterface 500 is displayed by the interface 134 of a client device 130.In some embodiments, the client device 130 may be a student device 130A,i.e., operated by a student user. In some embodiments, the client device130 may be a teacher device, 130B, i.e., operated by a teaching user. Insome embodiments, the user interface 400 may be displayed by a webpagebrowser, a mobile application, etc.

As shown in FIG. 5 , the user interface 500 illustrates the educationalcontent consumption behaviors of a user. The user interface 500 mayinclude a first interface element 510 and a second interface element520. The first interface element 510 may be associated with the user'sinteraction with the user interface 400 discussed in FIGS. 4A-4C. Forexample, the interface element 510 may includes the tags that are placedby the user to the corresponding portion of educational content. In someembodiments, the interactive element 510 may represent the user'sinteraction (e.g., tags) in an order that is associated with the timeand/or sequence of the user's interaction/action. The interface element520 may include a temporal graph illustrating the user's educationalcontent consumption behaviors in time. For example, the illustratededucational content consumption behaviors may include “app active,”“reading rate,” “page”, etc. The interface element 520 may present theuser's educational content consumption behaviors in a selected period oftime, e.g., form 9:00 am to 9:10 am. In some embodiments, theeducational content consumption behaviors may be recorded/presented inevery minute. In some embodiment, the layout (e.g., bar graph) anddisplayed parameters (e.g., time period, time interval, etc.) may bedetermined and/or customized by a user. In some embodiments, the userinterface 500 may be associated with a portion of educational contentand/or a specific user. In some other embodiments, the user interface500 may be associated with the whole educational content (e.g., a wholeclass session), and/or collective consumption behaviors of a set ofusers (e.g., a whole class of student).

In some embodiments, the user interface 500 may be presented to the userin real time as the user consume the content. In some embodiments, theuser interface 500 may be continuously updated. In some embodiments, theuser interface 500 may be presented periodically, i.e., after a certainamount of time. In one implementation, the user interface 500 ispresented after the user finishes the consumption of the whole targeteducational content, e.g., at the end of the lecture, at the end of achapter, etc.

FIG. 6A illustrates an example user interface 600, in accordance with anembodiment. The user interface 600 is a graphical user interface thatmay be provided by the interface 240 of the content server 110. The userinterface 400 is displayed by the interface 134 of a client device 130.In some embodiments, the client device 130 may be a teacher device 130B,i.e., operated by a teaching user. In some embodiments, the userinterface 600 may be displayed by a webpage browser, a mobileapplication, etc.

As shown in FIG. 6A, the user interface 600 presents learning analyticsassociated with the educational content. The learning analytics may beassociated the educational content consumption behaviors of a set ofusers. As shown in FIG. 6A, the educational content may include aplurality of portions, and the user interface 600 includes an interfaceelement corresponding to each portion of the educational content, e.g.,610, 620, 630, and 640. Each interface element may further include aplurality of interactive elements to present the learning analyticsassociated with the portion of educational content. For example, theinterface element 610 includes the interactive elements 612, 614, and616, which allow the user to interact with the interface element 610.

Referring to FIG. 6B, which illustrates an example user interface 600,in accordance with an embodiment. Following the context of the exampledescribed in FIG. 6A, FIG. 6B shows the user interface 600 changes toprovide one set of learning analytics after the user interacts withinteractive element 612. As shown in FIG. 6B, the interface element 610presents a statistic data of the educational content consumptionbehavior associated with a portion of the educational content. Forexample, the interface element 610 presents the percentage of users whoviewed the portion of educational content and the presentation of userswho input a response. In some embodiments, the interface element 610 maypresent the data learning analytics in graph. For example, the interfaceelement 610 illustrates the users' progress for consumption of theportion of educational content in a pie chart.

FIG. 6C illustrates an example user interface 600, in accordance with anembodiment. Following the context of the example described in FIGS. 6Aand 6B, the user interface 600 changes to provide another set oflearning analytics after the user interacts with interactive element614. As shown in FIG. 6C, the interface element 610 presents a statisticdata of the educational content consumption behavior associated with aportion of the educational content. For example, the interface element610 presents the time spent by the users on the consumption of theportion of educational content in a bar graph.

FIG. 6D illustrates an example user interface 600, in accordance with anembodiment. Following the context of the example described in FIGS.6A-6C, the user interface 600 changes to provide another set of learninganalytics after the user interacts with interactive element 616. In someembodiments, the interface element 610 presents a summary on one or moreeducational content consumption behaviors associated with a portion ofthe educational content. For example, the interface elements 610presents the users' interaction, such as, tag, comments, etc. in theuser interface 600 for a teaching user to review.

FIG. 7 illustrates an example machine to read and execute computerreadable instructions, in accordance with an embodiment. Specifically,FIG. 7 shows a diagrammatic representation of the data processingservice 102 (and/or data processing system) in the example form of acomputer system 700. The computer system 700 can be used to executeinstructions 724 (e.g., program code or software) for causing themachine to perform any one or more of the methodologies (or processes)described herein. The instructions may correspond structuring aprocessing configuration as described herein to execute the specificfunctionality as described with FIGS. 1 through 6 . In alternativeembodiments, the machine operates as a standalone device or a connected(e.g., networked) device that connects to other machines. In a networkeddeployment, the machine may operate in the capacity of a server machineor a client machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, a network router, switch or bridge,or any machine capable of executing instructions 724 (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute instructions 724 to perform any one or more of themethodologies discussed herein.

The example computer system 700 includes one or more processing units(generally processor 702). The processor 702 is, for example, a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), a controller, a state machine, one or moreapplication specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese. The processor executes an operating system for the computingsystem 700. The computer system 700 also includes a main memory 704. Thecomputer system may include a storage unit 716. The processor 702,memory 704, and the storage unit 716 communicate via a bus 708.

In addition, the computer system 700 can include a static memory 706, agraphics display 710 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), or a projector). The computer system 700may also include alphanumeric input device 712 (e.g., a keyboard), acursor control device 714 (e.g., a mouse, a trackball, a joystick, amotion sensor, or other pointing instrument), a signal generation device718 (e.g., a speaker), and a network interface device 720, which alsoare configured to communicate via the bus 708.

The storage unit 716 includes a machine-readable medium 722 on which isstored instructions 724 (e.g., software) embodying any one or more ofthe methodologies or functions described herein. For example, theinstructions 724 may include instructions for implementing thefunctionalities of the transaction module 330 and/or the file managementmodule 335. The instructions 724 may also reside, completely or at leastpartially, within the main memory 704 or within the processor 702 (e.g.,within a processor's cache memory) during execution thereof by thecomputer system 700, the main memory 704 and the processor 702 alsoconstituting machine-readable media. The instructions 724 may betransmitted or received over a network 726, such as the network 160, viathe network interface device 720.

While machine-readable medium 722 is shown in an example embodiment tobe a single medium, the term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 724. The term “machine-readable medium” shall also betaken to include any medium that is capable of storing instructions 724for execution by the machine and that cause the machine to perform anyone or more of the methodologies disclosed herein. The term“machine-readable medium” includes, but not be limited to, datarepositories in the form of solid-state memories, optical media, andmagnetic media.

Additional Considerations

The disclosed configurations beneficially provide a system (and/or amethod) for providing content recommendations. In some embodiments, thesystem applies a machine-learned comprehension model to behaviors of auser to determine a target measure of comprehension for each of aplurality of portions of target educational content. The system appliesa content identification model to identify supplemental educationalcontent and generates an educational content interface to present thesupplemental educational content to the user. In some embodiments, thesystem applies the machine-learned comprehension model to behaviors of aset of users to determine a collective measure of comprehension for eachof a plurality of portions of educational content. The system applies acontent identification model to identify a set of supplementaleducational content and generates a teacher interface to present the setof supplemental educational content. In this way, the system providesusers with an interactive and enriched learning experience.

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Embodiments according to the invention are in particular disclosed inthe attached claims directed to a method and a computer program product,wherein any feature mentioned in one claim category, e.g., method, canbe claimed in another claim category, e.g., computer program product,system, storage medium, as well. The dependencies or references back inthe attached claims are chosen for formal reasons only. However, anysubject matter resulting from a deliberate reference back to anyprevious claims (in particular multiple dependencies) can be claimed aswell, so that any combination of claims and the features thereof isdisclosed and can be claimed regardless of the dependencies chosen inthe attached claims. The subject-matter which can be claimed comprisesnot only the combinations of features as set out in the disclosedembodiments but also any other combination of features from differentembodiments. Various features mentioned in the different embodiments canbe combined with explicit mentioning of such combination or arrangementin an example embodiment. Furthermore, any of the embodiments andfeatures described or depicted herein can be claimed in a separate claimand/or in any combination with any embodiment or feature described ordepicted herein or with any of the features.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These operations and algorithmic descriptions, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as engines, withoutloss of generality. The described operations and their associatedengines may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software engines,alone or in combination with other devices. In one embodiment, asoftware engine is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described. The term “steps” doesnot mandate or imply a particular order. For example, while thisdisclosure may describe a process that includes multiple stepssequentially with arrows present in a flowchart, the steps in theprocess do not need to be performed by the specific order claimed ordescribed in the disclosure. Some steps may be performed before otherseven though the other steps are claimed or described first in thisdisclosure.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein. In addition, the term “each” used in thespecification and claims does not imply that every or all elements in agroup need to fit the description associated with the term “each.” Forexample, “each member is associated with element A” does not imply thatall members are associated with an element A. Instead, the term “each”only implies that a member (of some of the members), in a singular form,is associated with an element A.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the patent rights. It istherefore intended that the scope of the patent rights be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thepatent rights.

What is claimed is:
 1. A computer-implemented method, comprising:accessing, by a content server, historical consumption informationdescribing historical educational content consumption behaviors of a setof historical users and historical comprehension information comprisinga historical measure of comprehension of historical educational contentby the set of historical users; generating, by the content server, atraining dataset based on the accessed historical consumptioninformation and the historical comprehension information; training, bythe content server, a machine-learned model configured to predict ameasure of comprehension for a user consuming educational content basedon behaviors of the user as the user consumes the education content;applying, by the content server, the machine-learned model to behaviorsof a target user consuming target educational content to determine atarget measure of comprehension for each of a plurality of portions ofthe target educational content; and for a portion of the targeteducational content corresponding to a below-threshold target measure ofcomprehension: identifying, by the content server, one or morecharacteristics of the portion of the target educational content;applying, by the content server, a content identification model to theidentified characteristics of the portion of the target educationalcontent to identify supplemental educational content related to theportion of the target educational content; and generating, by thecontent server, an educational content interface to present thesupplemental educational content to the target user.
 2. The method ofclaim 1, wherein the educational content consumption behaviors includeone or more of reading rates, pause time, number of re-read times,delays in content consumption, highlighting, types of highlighting,highlight coverage, time of day, switch outs, and switch duration. 3.The method of claim 1, wherein the measure of comprehension includes oneor more of test scores, user evaluations of comprehension, types ofhighlighting, post-comprehension quiz results.
 4. The method of claim 1,wherein the content identification model is trained with a seconddataset comprising historical educational content consumed by historicalusers, and the historical educational content includes one or moretarget educational content and supplemental educational contentassociated with each target educational content consumed by thehistorical users.
 5. The method of claim 4, further comprising:accessing the second dataset; and training the content identificationmodel to predict a likelihood of supplemental educational content thatimproves a user's measure of comprehension for the portion of the targeteducational content.
 6. The method of claim 1, wherein applying themachine-learned model to determine a measure of comprehension comprises:determining the measure of comprehension in real time as the target userconsumes the portion of the target educational content.
 7. The method ofclaim 1, wherein generating the educational content interface to presentthe supplemental educational content to the target user comprises:displaying the supplemental educational content to the target user inreal time as the target user consumes the portion of the targeteducational content.
 8. The method of claim 1, wherein generating theeducational content interface to present the supplemental educationalcontent to the target user comprises: displaying the supplementaleducational content to the target user after the target user consumesthe portion of the target educational content.
 9. The method of claim 1,further comprising: identifying second supplemental educational contentrelated to a second portion of the target educational content, whereinthe second portion and the portion of the target educational content arerelated to a low-comprehension dedicated section; and modifying theeducational content interface to add the second supplemental educationalcontent for display.
 10. A computer system comprising: one or morecomputer processors; and one or more computer-readable mediums storinginstructions that, when executed by the one or more computer processors,cause the system to: access historical consumption informationdescribing historical educational content consumption behaviors of a setof historical users and historical comprehension information comprisinga historical measure of comprehension of historical educational contentby the set of historical users; generate a training dataset based on theaccessed historical consumption information and the historicalcomprehension information; train a machine-learned model configured topredict a measure of comprehension for a user consuming educationalcontent based on behaviors of the user as the user consumes theeducation content; apply the machine-learned model to behaviors of atarget user consuming target educational content to determine a targetmeasure of comprehension for each of a plurality of portions of thetarget educational content; and for a portion of the target educationalcontent corresponding to a below-threshold target measure ofcomprehension: identify one or more characteristics of the portion ofthe target educational content; apply a content identification model tothe identified characteristics of the portion of the target educationalcontent to identify supplemental educational content related to theportion of the target educational content; and generate an educationalcontent interface to present the supplemental educational content to thetarget user.
 11. The system of claim 10, wherein the educational contentconsumption behaviors include one or more of reading rates, pause time,number of re-read times, delays in content consumption, highlighting,types of highlighting, highlight coverage, time of day, switch outs, andswitch duration.
 12. The system of claim 10, wherein the measure ofcomprehension includes one or more of test scores, user evaluations ofcomprehension, types of highlighting, post-comprehension quiz results.13. The system of claim 10, wherein the instructions to apply themachine-learned model to determine a measure of comprehension comprise:determining the measure of comprehension in real time as the target userconsumes the portion of the target educational content.
 14. The systemof claim 10, wherein the instructions to generate the educationalcontent interface to present the supplemental educational content to thetarget user comprise: displaying the supplemental educational content tothe target user in real time as the target user consumes the portion ofthe target educational content.
 15. The system of claim 10, wherein theinstructions, when executed by the one or more computer processors,cause the system to: identify second supplemental educational contentrelated to a second portion of the target educational content, whereinthe second portion and the portion of the target educational content arerelated to a low-comprehension dedicated section; and modify theeducational content interface to add the second supplemental educationalcontent for display.
 16. A non-transitory computer-readable mediumcomprising stored instructions that when executed by one or moreprocessors of one or more computing devices, cause the one or morecomputing devices to: access historical consumption informationdescribing historical educational content consumption behaviors of a setof historical users and historical comprehension information comprisinga historical measure of comprehension of historical educational contentby the set of historical users; generate a training dataset based on theaccessed historical consumption information and the historicalcomprehension information; train a machine-learned model configured topredict a measure of comprehension for a user consuming educationalcontent based on behaviors of the user as the user consumes theeducation content; apply the machine-learned model to behaviors of atarget user consuming target educational content to determine a targetmeasure of comprehension for each of a plurality of portions of thetarget educational content; and for a portion of the target educationalcontent corresponding to a below-threshold target measure ofcomprehension: identify one or more characteristics of the portion ofthe target educational content; apply a content identification model tothe identified characteristics of the portion of the target educationalcontent to identify supplemental educational content related to theportion of the target educational content; and generate an educationalcontent interface to present the supplemental educational content to thetarget user.
 17. The non-transitory computer-readable medium of claim16, wherein the educational content consumption behaviors include one ormore of reading rates, pause time, number of re-read times, delays incontent consumption, highlighting, types of highlighting, highlightcoverage, time of day, switch outs, and switch duration.
 18. Thenon-transitory computer-readable medium of claim 16, wherein the measureof comprehension includes one or more of test scores, user evaluationsof comprehension, types of highlighting, post-comprehension quizresults.
 19. The non-transitory computer-readable medium of claim 16,wherein the instructions to apply the machine-learned model to determinea measure of comprehension comprise: determining the measure ofcomprehension in real time as the target user consumes the portion ofthe target educational content.
 20. The non-transitory computer-readablemedium of claim 16, wherein the instructions that when executed by oneor more processors of one or more computing devices, cause the one ormore computing devices to: identify second supplemental educationalcontent related to a second portion of the target educational content,wherein the second portion and the portion of the target educationalcontent are related to a low-comprehension dedicated section; and modifythe educational content interface to add the second supplementaleducational content for display.